Classical Single CPU and End Point Devices
FIG. 1 illustrates a classical single CPU network 100 in a PCIe switch domain. The uppermost PCIe device is a PCIe controller 102 of CPU network 100 with direct memory access to CPU memory 104. The uppermost device (PCIe controller) in this classical layout is called the “root complex” (root of the tree). Also illustrated down from the “root” are PCIe switches 106 and at end points are PCIe device endpoints 108. CPU network 100 uses a shared parallel bus architecture 103 for communication purposes, in which all devices share a common set of address, data, and control lines.
A CPU host 110 typically executes an operating system that determines the configuration of CPU network 100 by “enumeration”. Enumeration finds switches 106 and device endpoints 108 and claims total ownership over all the resources in CPU network 100. Type 1 and type 2 configuration read and write commands are used to discover switches 106, device endpoints 108, and memory resources owned by device endpoints 108 and switches 106. Memory addresses in a PCIe address space are assigned to all resources of both switches 106 and device endpoints 108. Base and limit switch registers are configured for all switches 106 to allow any point in CPU network 100 to understand how to route memory transactions around from device endpoint to device endpoint. CPU host 110 and device endpoints 108 can read and write memory resources anywhere in CPU network 100. Therefore, CPU host 110 can configure device endpoints 108 for use, and device endpoints 108 can act independently once setup to perform functions and read/write to and from CPU memory 104.
For example, one type of device endpoint 108 is an Ethernet controller 108a that can be setup to access transmit and receive descriptors from CPU memory 104, which point to buffers that CPU host 110 manages. Ethernet controller 108a can be associated with one or more PCIe devices (e.g., Ethernet devices). Each Ethernet device can independently transmit buffer contents described by transmit descriptors in its transmit descriptor ring (TX ring) and Ethernet controller 108a can receive data and send this data directly to receive buffers by its DMA function via a receive descriptor ring (RX ring) in CPU memory 104. According to Ethernet device programming and the PCIe standard, Ethernet controller 108a can send interrupts to CPU host 110 via a PCIe memory transaction known as an MSI/MSI-x message. This is a memory transaction that hits CPU host 110 device/interrupt registers in the CPU memory space mapped to PCIe.
Switches for this purpose are in wide use and the cost is relatively low. CPU network 100 is called a transparent PCIe network implementing a single PCIe domain. However, access to this PCIe domain from another network domain requires the use of specialized device drivers or specialized hardware.